Linear Convergence of SVRG in Statistical Estimation

SVRG and its variants are among the state of art optimization algorithms for the large scale machine learning problem. It is well known that SVRG converges linearly when the objective function is strongly convex. However this setup does not include several important formulations such as Lasso, group Lasso, logistic regression, among others. In this paper, we prove that, for a class of statistical M-estimators where strong convexity does not hold, SVRG can solve the formulation with a linear convergence rate. Our analysis makes use of restricted strong convexity, under which we show that SVRG converges linearly to the fundamental statistical precision of the model, i.e., the difference between true unknown parameter θ∗ and the optimal solution θ̂ of the model. This improves previous convergence analysis on the non-strongly convex setup that achieves sub-linear convergence rate.